MULTIPLICITY PRESERVATION FOR 
ORTHOGONAL-SYMPLECTIC AND UNITARY DUAL PAIR 
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Abstract. Over a non-archimedean local field of characteristic zero, we prove the 
multiplicity preservation for orthogonal-symplectic dual pair correspondences and 
unitary dual pair correspondences. 



1. Introduction 

Fix a non-archimedean local field k of characteristic zero, and a continuous invo- 
lution t on it. Denote by ko the fixed points of r. Then either k = ko or k is a 
quadratic extension of ko. Let e = ±1 and let E be an e-hermitian space, namely 
it is a finite dimensional k-vector space, equipped with a non-degenerate k -bilinear 
map 

(,) E :ExE^k 

satisfying 

(u, v) E = e(v , u) T E , (clu,v)e = cl(u,v)e, a E A, u,v E E. 
Write e' = — e, and let (E 1 , ( , ) E >) be an e'-hermitian space. Then 

E := E ® k E' 
is a k -symplectic space under the form 

(u ®u',v<g> v') E ■= tr k/ko ((u,v) E (u',v') E >)- 

Denote by 

(1) 1 - {±1} - §5(E) - Sp(E) - 1 

the metaplectic cover of the symplectic group Sp(E). Denote by 

H := E x k 

the Heisenberg group associated to E, whose multiplication is given by 

(u, t)(u', t') := (u + u',t + t' + (u, u') E ). 



Key words and phrases. Howe duality conjecture, oscillator representation, classical group, irre- 
ducible representation. 

1 



2 JIAN-SHU LI, BINYONG SUN, AND YE TIAN 

The group Sp(E) acts on H as automorphisms by 
(2) g.(u,t) := (gu,t). 



It induces an action of Sp(E) on H, and further defines a semidirect product Sp(E) x 
H. 

Fix a non-trivial character ip of k, and denote by ay, the corresponding smooth 
oscillator representation of Sp(E) x H. Up to isomorphism, this is the only genuine 
smooth representation which, as a representation of H, is irreducible and has central 
character if>. Recall that in general, if if is a group together with an embedding of 
{±1} in its center, a representation of H is called genuine if the element — 1 e if 
acts via the scalar multiplication by —1. 

Denote by G the group of all k-linear automorphisms of E which preserve the form 
(, )e- K is thus an orthogonal group, a symplectic group or a unitary group. The 
group G is obviously mapped into Sp(E). Define the fiber product 

G:=Sp'(E)x Sp(E) G, 

which is a double cover of G. Similarly, we define G' and G'. As usual, the product 
group G x G' is mapped into Sp(E). 

The goal of this paper is to prove the following theorem, which is usually called the 
multiplicity preservation for theta correspondences, and is also called the Multiplicity 
One Conjecture by Rallis in [Ra84j . It is complementary to the famous Local Howe 
Duality Conjecture. 

Theorem A. For every genuine irreducible admissible smooth representation n of 
G, and n' of G' , one has that 

dimHomg x g,(u;^, n <g> n') < 1. 

When the residue characteristic of k is odd, Theorem A is proved by Waldspurger 
in [Wa90] . The archimedean analog of Theorem A is proved by Howe in jHo89j . 

2. A GEOMETRIC RESULT 

We continue with the notation of the Introduction. Following [MVW871 Propo- 
sition 4.1.2], we extend G to a larger group G, which contains G as a subgroup of 
index two, and consists pairs (g, 8) G GLk (-E) x {±1} such that either 



5=1 and g G G, 
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or 

g(au) = a T g(u), a £ k, u £ E, and 

(gu, gv) E = (v, u) E , u,v E E. 

Similarly, we define a group G' and a group Sp(E), which extend G' and Sp(E), 
respectively. 

In general, if a group H is equipped with a subgroup H of index two, we will 
associate on it the nontrivial quadratic character which is trivial on H. We use xh 
to indicate this character. 

Denote the fiber product 

G:=Gx {±1} G' = {(g,g',5) | (g,S)eG, (g',5)eG'}, 

which contains 

G := G x G' 

as a subgroup of index two. Define a group homomorphism 

(3) ^ Sp(E), 

1 ' (g,g',S) » (g®g',5). 

Let Sp(E) act on the Heisenberg group H as group automorphisms by 

(4) (g,S).(u,t) := (gu,5t), 

which extends the action (T5]). By using the homomorphism £, this induces an action 
of G on H, and further defines a semidirect product 

J := G x H, 

which contains 

J := G x H 

ClS db subgroup of index two. 
Let the group 

(5) {±1} x (G x G) 
act on J by 

(6) (5,gi,g 2 ).j:=(giJg 2 ~y, 
where the semidirect product in d^J) is defined by the action 

-l.(gi,g 2 ) := (g2,gi)- 

The fibre product 

{±1} k {±1} (G x {±1} G) = {(5,gi,g 2 ) | Xc(gi) = Xg(&) = 5} 
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is a subgroup of (jSJ). It contains G x G as a subgroup of index two, and stabilizes J 
under the action (jSj). 

We prove the following proposition in the remaining of this section. 

Proposition 2.1. Every G x G-orbit in J is stable under the group {±1} X{±i} 
(Gx {±1} G). 

Let G act k -linearly on E by 
(7) (g, g' ,5).u <E> u' : = 5gu ® g'u' . 

Lemma 2.2. Every G-orbit in E is G-stable. 

We first prove 
Lemma 2.3. Lemma \2.2\ implies Proposition ^. 1[ 

Proof. Note that every G x G-orbit in J intersect the subgroup H, and the subgroup 
{±1} x {±1} (A(G)) of {±1} x {±1} (G x {±1} G) 

stabilizes H, where "A" stands for the diagonal group. Therefore in order to prove 
Proposition 12. 11 it suffices to show that every A(G)-orbit in H is {±1} X{±i} (A(G))- 
stable. Identify {±1} X{±i} (A(G)) with G. Then as a G-space, 

H = E x k, 

where E carries the action ([7]), and k carries the trivial G-action. This finishes the 
proof. □ 

Let G act k -linearly on 

E' := Hom k (E, E') 

by 

((j>,g',S).<f>)(u):=6g , {<f>(g r u)), 

where 

9 ' \ tg~\ if 5= -1. 
Then one checks that the ko-linear isomorphism 

E -> E', 

u®v! i — > (v (v , u) eu') 

is G-intertwining. Therefore Lemma 12.21 is equivalent to the following 
Lemma 2.4. Every G-orbit in E' is G-stable. 
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Denote by 

g := {x G End k (i?) | (xu, v) E + (u, xv) E = 0} 
the Lie algebra of G, and put 

g := {(x, F) \ x e g,F is a k-subspace of E, x\p = 0}. 

Let G act on g by 

(g,5).(x,F) := {5gxg-\gF). 
The action of G on E' induces an action of 

G = G/G' 

on the quotient space G'\E'. 

Lemma 2.5. There is a G -intertwining embedding from G'\E' into g. 
Proof. Recall that the map 

x h-> (x-, -) E 

establishes a ko-linear isomorphism form q onto the space of e'-hermitian forms on 
the k- vector space E. Define a map 

H:E' = Hom k (£,£') -> g, 

where F is the kernel of 0, and x is specified by the formula 

(<f>{u), 4>{v)) E ' = (xu, v) E , u,v E E. 

Use Witt's Theorem, one finds that two elements of E' stay in the same G'-orbit 
precisely when they have the same image under the map 5. Therefore 5 reduces to 
an embedding 

G'\E' - fl, 

which is checked to be G-intertwining. □ 

The following lemma is stated in |MVW87t Proposition 4.1.2]. We omit its proof. 

Lemma 2.6. For every (x,F) G g, there is an element (g, —1) G G such that 

gxg~ x = —x and gF = F. 

In other words, every element of g is fixed by an element of G \ G. Therefore 
every G-orbit in g is (5-stable. Now Lemma [2.51 implies that every G-orbit in G'\E' 
is G-stable, or equivalently, every G-orbit in E' is G-stable. This proves Lemma [2^1 
and the proof of Proposition 12.11 is now complete. 
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3. Proof of Theorem \M 

We first recall the notions of distributions and generalized functions on a t.d. 
group, i.e., a topological group whose underlying topological space is Hausdorff, 
secondly countable, locally compact and totally disconnected. Let if be a t.d. group. 
A distribution on H is defined to be a linear functional on C^(H), the space of 
compactly supported, locally constant (complex valued) functions on H. Denote by 
D^(H) the space of compactly supported distributions on H which are locally scalar 
multiples of a fixed haar measure. A generalized function on H is defined to be a 
linear functional on D^(H). 

Recall the following version of Gelfand-Kazhdan criteria. 

Lemma 3.1. Let S be a closed subgroup of a t.d. group H , and let a be a continuous 
anti- automorphism of H . Assume that every bi-S -invariant generalized function on 
H is a -invariant. Then for every irreducible admissible smooth representations n of 
H , one has that 

dim Honig(7r, C) ■ dimHonis(7r v , C) < 1. 

Here and henceforth, we use " v " to indicate the contragredient of an admissible 
smooth representation. Lemma 13.11 is proved in a more general form in SX08. The- 
orem 2.2] for real reductive groups. The same proof works here and we omit the 
details. 

Now we continue with the notation of the last section. 

Lemma 3.2. If a generalized function on J is G x G invariant, then it is also 
invariant under the group {±1} ^{±1} (G X{ ±1 } G). 

Proof. Note that the t.d. group J is unimodular. Therefore we may replace "general- 
ized function" by "distribution" in the proof of the lemma. Then by [BZ76, Theorem 
6.9 and Theorem 6.15 A], the lemma is implied by Proposition 12.11 □ 

Lemma 3.3. For every irreducible admissible smooth representations IT of 3 , one 
has that 

dim Hom G (II, C) • dimHom G (n v , C) < 1. 

Proof. The lemma follows from Lemma [37T1 and Lemma [3T21 by noting that an element 
of the form 

(-l,g,g)e{±l}x {±1} (Gx {±1} G) 
acts as an anti-automorphism on J. □ 

Let kty, 7r and tt' be as in Theorem A. As in the proof of [Su08t Lemma 5.3], 
c<A/, <8> vr v <g> 7r /v is an irreducible admissible smooth representation of J. Therefore 
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Lemma [3.31 implies that 

(8) dimHom G (wv ® 71_V ® vr' v ,C) • dim Hom G (wj ® 7T ® 7r', C) < 1. 

By |Su09l Theorem 1.4], the two factors in the left hand side of (JHJ) are equal. There- 
fore 

dim Hom G (u;,/, ® tt v ® tt /v , C) < 1 , 

and consequently, 

dimHomG(wf 7T (g) 7r') < 1. 
This finishes the proof of Theorem A. 
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